Repository navigation
Retry package-set heap exhaustion with clean compiler output - #815
Open
thomashoneyman wants to merge 2 commits into
Open
thomashoneyman wants to merge 2 commits into
thomashoneyman wants to merge 2 commits into
Conversation
thomashoneyman
commented
Oct 18, 2026
Member
Author
|
@f-f I deployed this branch before merging and reran the same 60-package update on the server, leaving the existing compiler output in place. It hit heap exhaustion, logged exactly one clean-output retry, and then succeeded. The run took about 20 minutes and published 81.8.0. No new out of memory kills, server restarts, or job resets during the run. So this has now handled the actual production failure, not just the local reproduction. |
f-f
force-pushed
the
fix/package-set-heap-retry
branch
from
October 18, 2026 22:57
a0b5e3c to
9c8503a
Compare
f-f
requested changes
Oct 18, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
So, we’ve been getting health check alerts because the compiler runs out of memory while building a package set with 60 updates. The previous change (#813) stopped those failures from taking down the server, but the build still hit the compiler’s memory limit. Surprising, since I had run the build locally in the same circumstances (so I thought) seen on the server!
Unfortunately, my error was to reproduce building the whole set from a clean state, whereas what happens in an actual package set build is that we build the current package set, install the updates, then compile again using the existing output. In a new local reproduction that failed with heap exhaustion. Removing only the compiler output and rebuilding the same sources succeeded within the memory budget.
The compiler loads existing compiled module information before deciding which modules dependency changes force it to rebuild, so an incremental build can have different memory costs from a clean one. I haven’t pinned down the exact source of the extra memory, but clearing the output consistently resolved this build’s failure.
So: in this PR, we detect the explicit heap-exhaustion message and allow one retry with clean output for package-set builds. For typical builds you get an incremental, fast one. For big builds that would exhaust memory, you get a retry. It keeps the rollback backup intact, leaves ordinary package builds unchanged. We were also misclassifying heap exhaustion as a "compiler race” (the unexpected JSON output bug).
The app tests pass. I also ran the full sequence through the updated handler under a 3.5 GiB memory limit. It hit the same error, cleared the output, and successfully rebuilt all 60 updates on its one retry, without using swap. So now I’m feeling confident this will work as expected.
The alternative as always is to host this somewhere with more RAM, but for now, this is a reasonable stopgap IMO.